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Executive Summary 

In recent years, community college leaders have begun 
to consider expanding the traditional role of institutional 
research (IR) at their colleges. This is due in part to several 
outside influences. Federal and state governments are 
pressing colleges to provide more data demonstrating 
evidence of student outcomes and institutional 
performance. Accreditation agencies are also asking 
colleges to provide evidence of student learning and 
achievement, and they want colleges to establish systems 
of institutional self-assessment to produce such evidence. 
The desire for more data and better analysis is also 
influenced by a growing enthusiasm among educators and 
advocates to use data to guide decisions about college 
management and about the design of college programs 
and services. This notion holds that data should be used 
not only for the purpose of accountability, but also for 
the explicit purpose of improving student outcomes and 
institutional performance. 

The Achieving the Dream: Community Colleges Count 
initiative, now in its third year of implementation, and 
several other national projects have also begun to raise 
awareness about the potential benefits of using data for 
improvement. Aided by a data facilitator and a coach, 
colleges participating in Achieving the Dream collect 
longitudinal data on their students (including courses 
taken, grades received, and programs completed), 
analyze the remedial and college-level course pathways 
taken by each cohort of students, and disaggregate the 
findings to determine if there are gaps in achievement 
among particular student groups defined in various ways, 
such as part-time students or students of color. Such 
analysis helps colleges to chart student progress over 
time and identifies points where some students tend 
to struggle or leave college entirely. Colleges can then 
develop strategies to improve student progress based on 
a clear diagnosis of the challenges that are present. 

Undertaking such an ambitious improvement strategy 
requires not only a capacity for data collection and 
analysis, but also a willingness to shift organizational 



culture in considerable ways. For such a strategy to 
be effective, stakeholders throughout the college must 
recognize and accept the legitimacy of internal research 
findings about student progress. Achieving the Dream 
recognizes the need for organizational transformation 
in creating such a climate and calls for college leaders 
to help nurture a “culture of evidence.” The process 
of building a culture of evidence requires the broad 
engagement of administrators, faculty, and student 
services staff in using data to understand where their 
students are experiencing problems, designing strategies 
for remedying those problems, and then evaluating the 
effectiveness of solutions implemented. It also involves 
institutionalizing the use of data analysis as the basis for 
program review, strategic planning, and budgeting. 

Community colleges that want to embark upon such 
a plan certainly face challenges. This report presents 
findings from a study conducted by the Community 
College Research Center (CCRC) on how well prepared 
today’s community colleges are in moving toward the 
greater use of data and research to improve student 
success. Data for the study were drawn from two major 
sources: first, an e-mail survey of a national random 
sample of 189 college administrators responsible for 
IR (1 1 1 full responses), and, second, case studies of 
28 community colleges in 15 states. The case studies 
included five or more in-depth interviews with college 
presidents, administrators, and faculty at each institution. 
The study aimed to learn how much IR capacity 
community colleges have in terms of IR staff size and 
facility with research methods, how IR is utilized by 
different actors within colleges, and what barriers may 
exist that impede the development of IR analysis that 
would benefit college decision making. 

Many persons who participated in the research are clearly 
aware of the growing interest in data-driven decision 
making. Indeed, most of the college presidents who were 
interviewed said that they had sought ways of improving 
data use at their college and had hopes of replacing 
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“institutional mythology” with fact. Yet, despite the interest 
and aspirations that were voiced at many colleges, results 
of the study reveal significant obstacles for strengthening 
the role of IR in improving college performance. 

The first challenge is a lack of research capacity found 
at many community colleges. IR offices are typically 
small and underfunded. In fact, about half of the colleges 
surveyed employ just one or fewer full-time equivalent 
(FTE) IR staff persons. In general, IR offices do not 
have the resources to take on more tasks beyond what 
is required by the time-consuming responsibilities of 
compliance reporting to federal and state agencies 
and preparation for accreditation visits. Tight staffing 
is especially prevalent among smaller colleges. The 
average FTE student enrollment at colleges with IR 
departments employing more than two FTE staff persons 
is 7,763. This is more than twice the average enrollment 
size of colleges with IR departments employing two or 
fewer FTE staff persons. 

Another obstacle in strengthening IR involves the 
difficulty that many colleges have in “cleaning” student 
data entered into student information systems at different 
times by multiple departments, and the difficulty in 
extracting these data from the same systems, which are 
not designed with research in mind. Without an easy- 
to-use data collection and analysis system, conducting 
the kind of research that could inform improvements to 
program and institutional performance is difficult. 

A final challenge focuses directly on organizational 
change. Among the leadership found at most community 
colleges, investing the resources and commitment 
required to build both a capacity for rigorous research 
and an atmosphere in which such research is valued and 
utilized is not yet a priority. Moreover, the study finds 
that administrators and faculty are often quite skeptical 
about the legitimacy of sophisticated studies in explaining 
student outcomes or informing improvements in teaching. 



While the findings call attention to the existence of these 
formidable challenges, they also shed light on how IR is 
structured at the small number of colleges in the study 
that do use data to manage and improve programs and 
services. Such colleges typically combine institutional 
research, planning, institutional effectiveness, and 
assessment into one department. These departments 
are led by an individual with experience and advanced 
training who is a full member of the college’s leadership 
team, and they employ sufficient staff to conduct 
research above what is required for the purposes of 
compliance and accreditation. Large IR departments 
such as these tend to be older than those found at other 
community colleges. Among surveyed IR departments 
that were formed prior to 1995, half employ three or more 
FTE staff persons. Among those departments formed 
since 1995, however, only 4 percent are this large. 

The study makes it clear that developing a sophisticated 
IR function and building a culture of evidence can 
take a long time. At one college advanced in its use of 
data for improving student outcomes and institutional 
performance, efforts in this direction began in the 
1980s and took hold during the 1990s. Administrators 
at that college say that it has taken them ten to fifteen 
years since then to establish a culture in which data- 
based decision making is institutionalized. During that 
time, each new president of the college has worked 
to preserve and extend gains made in terms of data 
collection, analysis, and college-wide research literacy. 

Community colleges that want to expand the traditional 
role of IR face several challenges in terms of resources, 
data collection, and institutional priorities. This study 
suggests that college leadership is a key component in 
making the necessary investment in IR capacity and in 
promoting changes in organizational practice that are 
required to embrace a strategy for using data to improve 
institutional performance and student success. 



Institutional Research and the Culture of Evidence at Community Colleges 



3 



Introduction 

The administrative area responsible for retrieving and 
analyzing student data in community colleges is typically 
the institutional research (IR) department. A 1999 survey 
conducted by Peterson et al. found that IR offices at 
two-year colleges were primarily responsible for preparing 
colleges for accreditation and meeting state reporting 
requirements. In recent years, however, community college 
leaders have begun to consider expanding the traditional 
role of institutional research at their institutions. This trend 
is largely driven by two developments: growing demands 
for accountability from policymakers and accreditation 
agencies and increasing recognition of the value of 
evidence on student progression for improving programs 
and services. 

Federal and state governments are pressing colleges to 
provide more data demonstrating evidence of student 
outcomes and institutional performance. Although most 
states have not required outcome of students tracked over 
time, the federal government has done so at least in a 
limited fashion with the development of the Student Right- 
to-Know performance measures. Implemented through the 
Integrated Postsecondary Education Data System (IPEDS), 
it requires colleges and universities to report graduation 
and transfer rates for cohorts of students over set time 
periods. Recently, the Secretary of Education’s Commission 
on the Future of Higher Education has stressed the need 
for improved data on college performance to enable 
prospective college students to make better informed 
choices. 

Accreditation agencies are also asking colleges to collect 
and analyze data by requiring evidence of student learning 
and achievement and the establishment systems of 
institutional self-assessment to produce such evidence. The 
standards of the New England Association of Schools and 
Colleges (NEASC) now call for the continuous collection of 
data on institutional effectiveness. The Higher Education 
Commission of the North Central Association of Colleges 
and Schools is encouraging the colleges it accredits 
to adopt an alternative accreditation process called 
Academic Quality Improvement Program (AQIP), which 
compels colleges to strengthen data collection and use the 
information to evaluate and improve practice. The Southern 
Association of Colleges and Schools (SACS) expects 



colleges to develop Quality Enhancement Plans that require 
the use of data in developing targeted efforts to improve 
student success and evaluating their effectiveness. 

The desire for more data and better analysis is also 
influenced by the growing enthusiasm for using data to 
guide decisions on college management and the design 
of programs and services (Petrides, 2004). This notion 
holds that data should be used not only for the purpose of 
accountability, but also for the explicit purpose of improving 
student outcomes and institutional performance. The 
Achieving the Dream: Community Colleges Count initiative, 
now in its third year of implementation, and several other 
national projects have also begun to raise awareness about 
the potential benefits of using data for improvement. Aided 
by a data facilitator and a coach, colleges participating 
in Achieving the Dream collect longitudinal data on their 
students (including courses taken, grades received, and 
programs completed), analyze the remedial and college- 
level course pathways taken by each cohort of students, 
and disaggregate the findings to determine if there are gaps 
in achievement among particular student groups, such as 
students receiving financial aid and students of color. Such 
analysis helps colleges to chart student progress over time 
and identifies points where students tend to struggle or 
leave college entirely. Colleges can then develop strategies 
to improve student progress based on a diagnosis of 
the problems and evaluate the strategies to inform 
improvements and guide allocation of resources to support 
practices that promote student success. 

Undertaking such an ambitious improvement strategy 
requires not only a capacity for data collection and analysis, 
but also a shift in organizational culture. Stakeholders 
throughout the college must recognize and accept the 
quality and legitimacy of internal research findings about 
student progress. They need to make key decisions about 
strategies and specific interventions based on evidence of 
what works for students generally rather than on anecdotes 
about individual student success. Achieving the Dream 
recognizes the need for organizational transformation in 
creating such a climate and calls for college leaders to help 
nurture a “culture of evidence.” The process of building 
a culture of evidence involves broad engagement of 
administrators, faculty, and student services staff in using 
data to understand where their students are experiencing 
problems, designing strategies for remedying those 
problems, and then evaluating the effectiveness of those 
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implemented strategies. Colleges participating in Achieving 
the Dream are expected to institutionalize the use of data 
analysis as the basis for program review, strategic planning, 
and budgeting. 

To better understand how well prepared today’s community 
colleges are to move toward greater use of data and 
research in improving student success, CCRC conducted 
a study of IR at community colleges. We sought to find out 
how much IR capacity community colleges have in terms 
of IR staff size, qualifications, and facility with research 
methods. We also asked what role and status IR has in the 
organizational structure of community colleges, how IR is 
used by different actors throughout colleges, and the extent 
to which it is currently used in decision making at different 
levels within colleges. Finally, we looked for common 
barriers that impede the development of the kind of IR 
analysis that could best inform decisions that affect student 
achievement. 

Study Sample and Method 

Data for our study were drawn from two major sources. 
First, we conducted an e-mail survey (spring 2005) of a 
national randomized sample of 1 89 college administrators 
with primary responsibility for institutional research (1 1 1 
full responses for a 59 percent completion rate; larger 
colleges were more likely to respond; no appreciable 
differences in urban/suburban/rural location or proportion 
of minority or part-time students between responding and 
non-responding colleges). Second, we conducted five or 
more in-depth interviews (with college presidents, senior 
academic affairs and student services administrators, 
institutional researchers, and faculty) about IR practices at 
each of 28 community colleges in 15 states to build case 
studies of each institution. By combining methodologies, 
we sought both a national perspective on institutional 
research practices, as well as a deeper and more nuanced 
understanding about the processes and structures of 
information management at individual colleges. 

Toward this end, case study colleges were selected to 
capture some of the variation in context that we perceived 
to be influencing IR practice at community colleges. The 15 
selected states represent five different levels (three states 
each) of involvement in performance accountability based 
on Burke and Minassians’ (2003) typology. Two colleges 



were then randomly selected from each state, one from 
institutions above the median size — based on full-time 
equivalent (FTE) enrollment — and one from those below 
the median size. In two states, we were unable to recruit 
any of the state’s small colleges to the study, resulting in 28 
case studies across the 15 states. 

Our interviews focused on how IR is perceived and utilized 
by different stakeholder groups within an institution that 
work directly with issues related to students and student 
success, namely academic affairs, student services, and 
faculty. We also wanted to examine the role of leadership 
and organizational reporting structures that determine 
how IR is implemented, making the interviews with each 
college’s president key to our analysis. 

Findings 

Characteristics of Community 
College Institutional Research 

Staffing. Our survey and case study results indicated a 
wide range of IR staffing patterns. These structures fell 
into four basic categories. First, there were those colleges 
with no IR function at all. Three colleges (3%) participating 
in our survey fell into this category. Second were those 
colleges at which an IR function exists, but less than 1 
full-time equivalent (FTE) staff member is devoted to it. 
Fifteen colleges in our survey (14%) structured IR in this 
way. In these cases, research made up about one third to 
one half of an administrator’s responsibilities. Respondents 
at 6 of these 15 colleges were in senior positions (dean, 
vice president, or higher). The remaining colleges in our 
survey were evenly split between the final two categories. 
Forty-five colleges (41 %) employed a single IR director. 
Some of these IR directors reported having minor part- 
time assistance from students or administrative assistants 
totaling 0.5 FTE or less. At 18 of the colleges falling 
into this category, the lone IR director reported directly 
to the president. The final category consisted of those 
colleges with IR departments employing more than 1 .5 
FTE researchers. This category contained 48 (43%) of our 
respondent colleges. 

IR directors (as opposed to senior administrators with IR 
responsibilities) fall very much into the middle management 
range of community college organizational structure. This 
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tends to be true even when an IR director reports directly to 
the college president. Their “director” status does not afford 
them the level of responsibility or authority that would be 
given to a vice president. 

The number of full-time equivalent (FTE) IR staff at the 
surveyed colleges ranged from 0 to 7. Nearly three quarters 
of the surveyed colleges had two or fewer FTE staff 
devoted to IR, and just over half of these colleges actually 
had one or fewer. We estimate that one fifth of community 
colleges have little or no IR capacity beyond very basic 
reporting functions due to very limited staffing. Our case 
study colleges showed a similar range of staffing patterns. 

Two key factors that appear to relate to the size of an I R 
department are the size of the college and the age of the 
department. Our survey showed that the average student 
FTE for departments with more than two full-time IR staff 
was 7,763. This is more than twice the size of the average 
for colleges with 2 or fewer FTE researchers. Larger IR 
departments also tended to be older. Among departments 
formed prior to 1 995 (there were 39 of these, or just under 
half of those with departments responding to this item), 51 
percent were large (3 or more FTEs) and 1 3 percent were 
small (fewer than 1 FTE). Conversely, among departments 
formed 1 995 to present, 4 percent were large and 47 
percent were small. 

These results are important because they have implications 
for creating the kind of data environment that is necessary 
for building a culture of evidence. The colleges that are 
likely to need the most help in developing their research 
capacity to use data to improve student outcomes are 
those that are medium to small in size. Those (mostly 
larger) colleges with the most capacity for these functions 
represent a relatively small subset of community colleges 
nationwide. 

We found that some colleges used grant funds to support 
research beyond basic compliance reporting. And our 
case studies revealed another potential strategy for 
increasing IR capacity. Some of the IR departments that 
devoted more time to analytical research were more likely 
to employ graduate students to do this kind of work. 

Further inspection of the survey data, as well as some case 
study interviews, supported the finding that community 
colleges situated near research universities have more FTE 
researchers than do those without a nearby university. 



Preparation of institutional researchers. The preparation 
of IR personnel is an important topic that bridges the 
organizational structure of IR with the types of research 
conducted. About half (54 percent) of senior IR personnel 
responding to our survey had been working in IR for six 
or more years. Based on our case study findings, there 
appears to be considerable movement in IR personnel, 
with many of our interviewees reporting that they were new 
to their colleges, but not to their occupations. No typical 
pathway to working in IR was evident in our interviews. Only 
one individual had earned a degree that explicitly prepared 
her for work in an IR office. Several interviewees described 
starting out in some other department in the college and 
getting pulled into IR because they were either looking for 
more challenging work or were predisposed to working with 
numbers. 

Only 1 1 percent of our survey respondents held less than 
a master’s degree. About half held master’s or professional 
degrees (51 percent). Nearly 40 percent (38 percent) 
held either a PhD or EdD. Most of these degrees were in 
education (32 percent), followed by the social sciences 
(23 percent), and business (17 percent). A number of 
respondents held two equivalent degrees (for example, 
an MS in policy analysis and an MA in political science); 
frequently these included a background in policy or urban 
studies. 

We asked how many semesters of coursework respondents 
had completed in quantitative methods. Such coursework 
was evenly distributed over the group as a whole, with 
about half of the respondents having three or fewer 
semesters and half having four or more. However, senior IR 
staff holding doctorates were about twice as likely as others 
to have completed four or more semesters in quantitative 
methods (only nine percent of survey respondents held 
social science doctorates with four or more courses 
in quantitative methods). Individuals with education or 
business degrees tended to have fewer semesters of 
quantitative coursework. Our interview data were consistent 
with these survey findings. 

The question raised by these differences in educational 
preparation is whether the background of an IR director 
appears to make any difference in the way that IR is 
conducted and used at the colleges. We hypothesized 
that IR directors with PhDs, and particularly those in the 
social sciences who have considerable training in research 
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methods, may be conducting more sophisticated research 
at their colleges, and that their research may be more 
likely to be intended for organizational decision making. 

This hypothesis did not hold true based on our survey and 
interviews. While highly qualified IR personnel are more 
likely than others to use sophisticated methodologies, 
it is not clear that these methodologies are considered 
necessary for the job. Indeed, the more sophisticated IR 
work may be done for the purpose of journal submissions 
and conference presentations rather than for college 
management. In general, we found examples of strong and 
weak institutional research used for college management 
and improvement of programs and services regardless 
of the backgrounds of IR leadership. We discuss this at 
greater length in sections that follow. 

Sources of data for IR. The data used by IR come from a 
variety of sources. Our survey results indicated that state 
and IPEDS data (which originally come from the colleges 
themselves) are two of the most important sources of 
information about students, followed by national student 
surveys such as the Community College Survey of Student 
Engagement (CCSSE) and Noel Levitz, followed by 
homegrown surveys. 

IR departments make use of college-based student 
information systems, which are simultaneously an excellent 
source of data and the most significant barrier to data 
use. Student information systems are made up of all the 
information that college personnel enter into their databases 
about students. This process begins with a student’s 
application and continues throughout her educational 
experience at the college. But student data are entered 
by a wide range of college personnel, and in some cases 
by work-study students or other part-time help who are 
hired for data entry, but are sometimes not trained to do 
so effectively. Therefore, if the data in the system are to 
be used for research purposes, one of the first roadblocks 
encountered by IR offices trying to expand their research 
capacities is the need to crosscheck student information 
and make sure that the processes in place for entering data 
include procedures for identifying and correcting errors. 
Student data are entered by many college departments. 
Typically, admissions, financial aid, counselors, placement 
test administrators, and faculty all have their own interfaces 
with a college’s student information system. In some 
cases, there are areas of the database that do not interact 
with each other. This is the case with one frequently 



used system, which consists of separate subsystems for 
students, financial aid, finance, human resources, and 
advancement. This compartmentalization can create hours 
of extra programming if data from more than one sub- 
system are needed. 

Although student information systems are rich sources of 
data, they are not designed with research in mind; rather, 
they are designed primarily to serve student record- 
keeping purposes as well as other management functions 
such as college financial administration, facilities use, 
and class scheduling. It is possible to run “data extracts” 
from these systems consisting of individual student data 
that can be imported into programs such as Microsoft 
Access, Excel, SPSS, or SAS for research purposes. 

Our interviews show that at many colleges only a small 
number of administrators and faculty have been trained to 
access data from student information systems for analysis. 
The lack of easy accessibility to the vast store of data 
available at community colleges is a major challenge for 
many community college IR departments. At a minimum, 
it creates a bottleneck situation, where IR staff spend time 
accessing and organizing data for their own purposes or 
assisting others on campus with these tasks. 

Both our survey and case studies suggest that student 
data collected and archived by state community college 
or higher education agencies are used by IR personnel. 

Our survey asked respondents whether they had access 
to a state-level data warehouse. Three fifths (59 percent) 
of respondents reported having access to state data 
warehouses, over half (56 percent) of respondents reported 
having access to college data warehouses, and 18 percent 
to district data warehouses. Overall, 42 percent of the 
survey respondents both had access to a state data 
warehouse and used it “frequently.” Our interview results 
conflicted with this information, however. We learned that 
while data may be archived at the state, district, and college 
levels, these data are only used by the most advanced IR 
departments. IR personnel did tell us they routinely use 
data reports published by state and district offices. These 
reports are generally descriptive and include information on 
enrollment patterns disaggregated by a variety of student 
characteristics, including enrollment status, race, gender, 
and program or degree. 

Many IR departments also collect data through surveys. 
Surveys of students who have left the college are the 
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most common. Response rates are often extremely low, 
but the information can provide some indication to the 
college about the types of work and further education 
students pursue after graduation. Employer surveys are 
also frequently used, often as part of the program review 
process for occupational programs. Businesses that are 
known to employ community college graduates are asked 
whether they are satisfied with the levels of preparation 
of students. Large scale surveys are also conducted with 
enrolled community college students in order to assess 
student satisfaction or “engagement” in the college. These 
tend to be national surveys, such as the Community 
College Survey of Student Engagement (CCSSE) and Noel- 
Levitz. 

In addition to these routine types of surveys, IR 
departments are sometimes involved in conducting surveys 
on behalf of faculty and staff; often the IR staff assist in 
writing and designing the survey and turn over the task 
of analyzing the data to whomever requested the survey. 
Such surveys are conducted for a wide range of reasons: 
sometimes in preparation for grant proposals, sometimes 
as part of an existing grant, and sometimes because faculty 
and staff are pursuing degrees and credentials themselves 
that have a research requirement. Finally, IR departments 
occasionally conduct focus groups. Our interviews suggest 
that these are relatively rare; they are typically carried out by 
IR departments that are somewhat more sophisticated in 
their approach to research. 

Uses of Institutional Research 

State and federal compliance reporting. Much of the 
time of IR staff is devoted to reporting data to a variety 
of external stakeholders, especially state and federal 
government agencies. The nature of data requests coming 
from states varies widely and depends on the structure of 
the state system and methods of collecting and organizing 
data. Some state systems are more centralized, with 
significant research capacity at the state level, while others 
have little or no statewide community college infrastructure. 
Centralized state systems may or may not have centralized 
data systems. In most instances, each time the state 
system needs particular information regarding student 
enrollments, it must request a report from each of the 
colleges in the system. In states such as these, something 
as simple as reporting the percent of female community 
college students statewide requires each college to report 



counts of students disaggregated by gender. This kind of 
activity takes up much of the time of IR offices. 

Federal reporting is similar and is generally associated 
with the Integrated Postsecondary Education Data System 
(IPEDS) and grant-funded programs, such as those under 
the Perkins Act, and Titles III and V. While it might sound 
as if state and federal reporting are redundant, they 
are not, because each data point reported has its own 
parameters, and there is often no coordination between 
state and federal agencies on data definitions. Based on 
our interviews, it is commonly believed that the demands 
of state and federal compliance reporting have been 
increasing in recent years. This is not surprising, given the 
current climate of increasing accountability. 

For the purposes of this report, we describe federal, 
state, accreditation, and grant reporting as “compliance 
reporting.” An important subset of compliance reporting 
involves performance accountability. Some states have 
implemented performance accountability systems for 
community colleges. Performance accountability reporting, 
in which colleges report on such indicators as rates of 
retention, transfer, and graduation, job placement success, 
student performance on licensing exams, and student 
satisfaction, differs from other compliance reporting in that 
theoretically the stakes are higher. Sometimes states simply 
report the data to external audiences, but in a few cases 
financial rewards or sanctions are determined based on 
performance. Our research design took into consideration 
the demands of state accountability systems, with the 
hypothesis that the degree of performance accountability 
required of community colleges might be related to IR 
capacity. We found that this hypothesis was not supported 
by either the survey or the case study data. In fact, the most 
salient characteristic associated with greater IR capacity 
was college size, regardless of the state in which the 
colleges were located. 

Colleges did not perceive the time required to keep up 
with compliance reporting as well spent. For example, one 
vice president observed that “the federal burden alone 
consumes probably 20-25 percent of the IR office, and 
that is basically data of no use to us. The results of it don’t 
inform any of our decision making and are so aggregated 
that they are not relevant to the decisions that we make 
on campus.” In general, there was a consensus that the 
information generated in response to most compliance 
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reporting requests, while sometimes indirectly used 
in college decision making, did not really contribute 
significantly in learning about students. And faculty at 
many colleges questioned new demands by accreditation 
agencies for greater accountability reporting. They 
expressed doubts about the usefulness of such measures 
in informing faculty for purposes of actual program 
improvement. 

A central problem with the responsibility of compliance 
reporting is that it takes time away from carrying out 
research that could play a role in college operations, such 
as examining the rate at which remedial students go on to 
enroll and succeed in college-level, degree credit programs 
and what practices are effective in increasing that rate. 

From the standpoint of the colleges, the state and federal 
compliance reporting role of IR is non-negotiable since 
they have no choice but to participate. Therefore, colleges 
are compelled to allocate personnel time to this function 
rather than to other types of research. Since reporting 
requirements are more or less the same across all colleges 
within states, regardless of enrollments, the impact of these 
demands may be relatively greater on smaller colleges. A 
college that is large enough to support an IR department 
with several staff can devote a smaller percentage of its 
overall IR resources to this function. 

Internal reporting. In addition to external compliance 
reporting, internal reporting is the other main responsibility 
of IR departments. The data most frequently requested 
by college administrators concern enrollments. At 
approximately one third of our case study colleges, 
enrollment reports were produced daily or weekly and 
circulated to college administrators, including deans and 
department chairs. At most of the remaining colleges, these 
reports were circulated at critical times in the semester. 
Based on our interviews, the explanation for this focus on 
enrollment monitoring is that in most states colleges are 
funded primarily on the basis of student enrollments, so the 
only way that budgets that have been created in the spring 
can be successfully supported in the fall is if enrollments 
meet or exceed projections. If they do not, deans and 
department heads need to find ways to shuffle adjuncts 
and close sections so that the college will not exceed the 
budget. 

The primary internal audiences for IR, according to both 
our survey and interview results, were college presidents 



and other college administrators. Over four fifths of survey 
respondents described college presidents and vice 
presidents (84 percent) and college administrators (86 
percent) as frequent recipients of reports, compared to 45 
percent describing faculty as frequent recipients. Only 1 8 
percent considered trustees to be frequent recipients. 

The fact that college presidents were often perceived by 
IR staff as a primary audience for their work deserves 
attention. Many of the IR personnel in our case studies 
told us that it was extremely important to them to report 
directly to the college president as opposed to reporting 
to academic affairs, information technology, or some other 
division. Two reasons were given for this. First, reporting 
directly to the college president elevates the status of IR 
so that data are more likely to be used in decision making. 
In a large number of such cases, IR is included on the 
president’s cabinet. At the same time, even though data 
may be used in decision making, IR directors themselves 
generally do not have the level of responsibility or 
authority that would be given to a vice president. Among 
those colleges with newly established IR departments, 
our interviews indicated that responsibility for IR was 
shifting upward in the organizational hierarchy from an IR 
“coordinator” or part-time role to director. In colleges with 
established IR departments we found that senior positions 
were being created that merged IR with strategic planning. 
Second, reporting to the college president allows IR to 
be used college-wide rather than becoming beholden to 
certain departments or functions. 

It is important to emphasize that faculty were much less 
likely to be perceived as frequent recipients of IR reports. 

In general, faculty were reported to be not very involved 
in IR, either as an audience or as participants in research. 
According to our surveys and interviews, when faculty 
involvement in IR occurred, it was most often in the context 
of strategic planning or the accreditation process. 

Through our case studies, however, we did hear about 
interesting studies conducted by individual faculty 
members or groups of faculty from the same program 
with the assistance of IR. These studies tended to focus 
more on student outcomes and pathways through college 
than did the routine research carried out by IR personnel. 
One college described a recent example of a study 
looking at whether gains in writing skills of students taking 
composition early on in their college careers are sustained 
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in their later course taking. IR worked with faculty to assess 
the writing skills of their students, and then disaggregated 
the results according to when the students had taken 
composition. At another college, the math department 
wanted to know about the effectiveness of its “tech math 
class.” In particular, the math faculty wanted to know how 
students taking this class fared once they reach pre- 
calculus. The study confirmed that students taking “tech 
math” were doing as well as others when they reached 
pre-calculus. This department also used IR to look at the 
success rates of students starting in developmental math 
and found that these students had better rates of success 
in college-level courses than students who had not taken 
the developmental courses. 

IR priorities. While these examples illustrate that 
individuals and groups within colleges occasionally utilize 
IR to carry out small studies about student progress, the 
majority of IR work relates to compliance reporting and 
enrollment monitoring. Applied research that is either 
analytical or evaluative is less common and is considered to 
be a lower priority. 

We asked colleges about other priorities for IR, what other 
types of studies they conduct, and what methodologies 
they use. In addition to preparing the reports requested 
by state and federal agencies, at three quarters of the 
colleges surveyed, preparation for accreditation visits was 
considered a high priority for IR offices. At a minimum, this 
involves ongoing program review and generating data for 
accreditation reports. This finding was consistent with our 
case studies. Time spent preparing for accreditation goes in 
cycles. It becomes the focus of IR offices for a year to two 
years prior to a visit, which means a realignment of priorities 
occurring every five or ten years. 

Program review, which is an ongoing part of the 
accreditation process, was a high priority for half of the 
survey respondents. IR staff are generally responsible 
for pulling together basic factual information about the 
college such as enrollment patterns and demographic 
characteristics of students by program. Our case studies 
indicated that ongoing program review often includes 
other items of information, such as employer surveys and 
graduation statistics. Some colleges are being pushed 
toward more sophisticated processes to replace program 
review that will eventually include measurements of student 
learning. 



Student retention studies were reported to be a high priority 
by over half of the colleges, but this finding was not well 
supported by our case studies. Instead, our interviews 
indicated that, while studies of student retention are done 
by IR departments at some colleges, they were not given 
high priority at any of our case study sites. It is possible that 
this discrepancy reflects differing definitions of retention. 
Retention can, for example, be studied from semester to 
semester and year to year. But based on our interviews, 
community colleges often associate “retention” with 
the rate at which students pass individual courses, not 
persistence across terms or advancement to higher-level 
courses. We believe that many respondents to our survey 
were thinking of course pass rates when answering the 
question about the priority for studies of retention. 

It is important to recognize that in order to analyze what 
actually happens to students enrolling in college, it is 
necessary to identify cohorts of students and use student 
unit data to follow their progress over extended periods of 
time. Our interviews suggest that longitudinal studies of 
student progression are rare at community colleges. Even 
IR departments with the most capacity conducted such 
studies only once every few years. 

Transfer is one area of longitudinal research that does 
appear to receive attention. When we asked colleges about 
following student progress over time, many interviewees 
described studies they routinely conducted in which they 
seek information about what happens to students once 
they transfer to four-year universities. Some states routinely 
use National Student Clearinghouse data to identify 
students who transfer into the state university system. In 
states where these systems do not exist, we were given 
examples of arrangements to obtain data that had been 
made between specific institutions. For example, several 
community colleges in our study reported following up 
with their transfer students at key receiving institutions. 

In some instances, these follow-ups involved providing 
data on student course completions, but in several cases 
we learned that IR offices had conducted focus groups 
or surveys with students who had transferred. In addition, 
many community colleges are beginning to pay a per- 
student-tracked subscription fee to use National Student 
Clearinghouse Data, which is a source of data linked to 
financial aid that allows colleges to track students who 
leave their institutions and go on to enroll in other colleges 
and universities. 
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We also sought information about the extent to which 
community colleges conduct studies in which student 
data are disaggregated to learn about the differences 
between groups of students sharing common background 
characteristics, such as race and ethnicity, full- or part- 
time attendance, age, gender, and parents’ education. 

Only 26 percent of survey respondents considered studies 
of student outcomes disaggregated by race/ethnicity to 
be high priority; indeed, 41 percent reported that these 
studies are either not done at all or given low priority. Our 
interview findings were consistent with these results. Few 
IR offices produced reports disaggregating students by 
any variable; those that did so were most likely to use race 
and ethnicity. College officials offered explanations for why 
disaggregating by race and ethnicity was not done. For 
example, at a large New England college, we were told 
it would not be productive because they could not tease 
out differences within standard racial or ethnic categories. 
Institutional researchers at a small, predominantly white 
college reported that they do not disaggregate by race 
because it would be too easy to identify students. 

Longitudinal analysis and the study of relationships 
between student backgrounds and outcomes require some 
sophistication in research methodologies. Multivariate 
analysis is needed if a researcher seeks to control for the 
impact of student background in equations predicting 
outcomes, or wants to use a sample of students to analyze 
the outcomes of a larger student population. In our survey 
and interviews, we asked researchers to tell us about the 
methodologies they use to conduct IR studies. The survey 
results show that almost all of the respondents reported 
conducting basic descriptive analysis of data — for 
example frequencies (89 percent) and rate computations 
(100 percent). However, with the exception of correlations 
(57 percent), fewer than half reported using any kind of 
statistical technique (for example, 48 percent use chi square 
tests, 40 percent use linear regression, and 22 percent use 
logistic regression). These results are consistent with our 
interviews with IR directors, in which few used anything 
beyond rates, frequencies, and cross-tabulations unless 
they sought to publish their work in a professional journal. 

The results of our survey indicate that the types of 
methodologies used by researchers are related to their 
educational backgrounds. Researchers holding MAs and 
PhDs reported using more of the complex methodologies 
(19 percent and 32 percent, respectively, compared 



with 0 percent for BA holders), with a clear increase in 
complex methodology use (defined as time-series analysis, 
multinomial logistic regression, canonical correlations, 
path analysis, or event history modeling) among PhDs. 
Respondents with degrees in the social sciences or 
in fields such as public policy or public administration 
or urban studies were also more likely to use complex 
methodologies (30 percent and 25 percent, respectively). 
Consistent with these results, researchers using complex 
methods were more likely than others to have four or 
more semesters of quantitative coursework (31 percent 
versus 14 percent for 0-1 semesters and 10 percent for 2-3 
semesters). Performing longitudinal analysis using student 
cohort data, which is essential for well-conceived data- 
based decision making, requires higher-level knowledge 
of methodologies and statistics. Our research on IR 
departments suggests that at many colleges the capacity 
for this kind of analytical research does not exist. However, 
even in those instances where it does exist, IR personnel 
rarely, if every, carry out sophisticated analyses for use in 
college management or in efforts to improve programs 
and services. This may explain the finding from our study 
that respondents generally perceived that knowledge of 
advanced statistics is not essential to performing in the role 
of institutional researcher. 

Our research revealed a cultural resistance to complex 
research. In our interviews with community college 
personnel, we were repeatedly told by college presidents, 
vice presidents, and institutional researchers themselves 
that complex studies were not useful to them. Two major 
explanations for this emerged. One involves the legitimacy 
of complex studies for learning about community college 
students. We were told by our interviewees that there are 
simply too many variables to account for when analyzing 
the educational outcomes of community college students. 
The second was the perception that there is simply 
no audience for complicated analysis. These findings 
dovetail with the widespread faculty resistance to greater 
accountability demands in accreditation that was found to 
exist at many colleges. 

Perceived Barriers to Effective Use of 
Institutional Research 

To better understand the barriers confronted by IR at 
community colleges, our survey asked respondents to 
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check off all applicable responses to the following question: 
“Which of the following would you need in order to increase 
the effectiveness of institutional research at your college?” 
The need for additional staff topped the list, with most of 
the respondents (85 percent) describing this as a problem. 

In addition, a fifth of the respondents reported that they 
are hampered by the absence of an IR department, which 
also suggests problems with staffing resources. Along the 
same lines, the need for professional development for IR 
staff was identified by more than half (56 percent) of the 
respondents. Nearly a third of the respondents (31 percent) 
told us that they needed upper-level college administrators 
to use institutional research in order for them to become 
more effective. 

Thus it is not surprising that we found only a small 
number of colleges collecting data on student outcomes 
for purposes of strategic planning and program 
improvement. The IR departments that were most 
advanced in using data for college management were 
those that combined research, planning, institutional 
effectiveness, and assessment into one department. 
These departments were generally led by an individual 
with a PhD who was a full member of the college’s 
management team, and employed staff with a range of 
job roles to fit the many requirements of managing and 
analyzing data. This differentiation of roles streamlines 
the process of responding to external demands, giving 
the senior leadership of the department more time to 
design and implement studies of internal importance to 
the college. At these colleges, which tended to be large 
in size, there is sufficient IR capacity to produce research 
that is respected and relevant to institutional management 
and efforts to improve teaching and learning. 

Conclusion 

Many of our case study colleges were clearly aware of the 
growing interest in data-driven decision making. Most of 
the college presidents we interviewed told us that they 
sought ways of improving data use at their colleges and 
had hopes of replacing “institutional mythology” with 
fact. We were told about a number of external pressures 
that were helping to elevate the importance of data. 
Accreditation teams are telling colleges that their capacity 
for institutional research and assessment of learning 
needs to be improved. State governments are also 



increasing demands on colleges for data, particularly as 
part of performance accountability systems. And we also 
heard from some colleges that the League for Innovation’s 
Learning College initiative had encouraged them to place 
a greater priority on data-based decision making. 

Yet, our study shows that by far the most common 
uses of institutional research by community colleges 
are compliance reporting to federal and state agencies 
and preparation for accreditation visits. And there was 
a consensus among respondents that the information 
generated for compliance reporting and even accreditation 
is not very useful to college management or faculty and 
student support staff. Compliance reporting takes time 
and resources away from research that could benefit 
college operations and help to improve student outcomes. 
Internally, institutional research is most often used to 
monitor enrollments. Few colleges systematically track 
student progress and outcomes over time, and even fewer 
use this information to improve programs and services. 

Our study reveals some formidable challenges to 
strengthening the role of institutional research in building 
a culture of evidence at community colleges. The first is 
a lack of research capacity at many community colleges 
to do more than what is required beyond the time- 
consuming responsibilities of compliance reporting and 
preparation for accreditation. About half of the colleges 
we surveyed have one or fewer full-time IR staff persons. 
We estimate that about one fifth of colleges have little 
or no IR capacity beyond very rudimentary reporting 
functions due to limited staff (often less than one full- 
time person) and, in some cases, a lack of training and 
experience on the part of IR staff. 

Colleges with larger IR staffs (which tend to be the 
larger institutions) and those that had established IR 
departments for a long time were able to free up staff 
from compliance reporting to conduct research related to 
the management and practice of teaching and learning. 
Some colleges used grant funds to support research 
focused on improvement while a few recruited graduate 
students from nearby universities. 

The difficulty many colleges have in “cleaning” student 
data entered at different times by multiple departments 
and the difficulty in extracting these data from student 
information systems designed to support administrative 
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functions rather than research are two other impediments 
to doing research that could inform improvements to 
program and institutional performance. 

We also found that, to a large extent, a college’s president 
determines the way research and data are used in the 
management of the institution. The IR administrators we 
surveyed most often perceived college presidents as 
the primary audience for their work. As mentioned, our 
interviews revealed a clear awareness among presidents 
of the greater pressures for accountability and of the 
potential value of using data for improvement. Yet, in only 
a few cases are community colleges using longitudinal 
data on student progress and success — as opposed 
to using only data on enrollments — to manage the 
institution. This may reflect the skepticism reflected 
in our interviews with college presidents (as well as 
other college personnel) about the legitimacy of more 
sophisticated studies in explaining community college 
student outcomes. It may also mean that some presidents 
do not know how to use data to manage the institution. 
This would suggest the need for more education on data- 
based management in doctoral programs that prepare 
college presidents and in continuing education. 

Presidents who want to move their institutions toward 
greater use of data to improve programs and practices 
may well face resistance from the faculty. Some of the 
faculty we interviewed questioned whether institutional 
research could provide information useful in improving 
teaching. Many faculty expressed doubt that the 
increased accountability reporting being demanded by 
accreditation agencies would lead to improved outcomes 
for students. IR staff at some colleges work with 
individual faculty to conduct studies of the effectiveness 
of particular programs or strategies, but these efforts 
tend to be isolated. IR staff considered faculty a primary 
audience for their work at fewer than half of the colleges 
surveyed. 

The IR departments in the small number of colleges we 
found that did use data to manage and improve programs 
and services combined research, planning, institutional 
effectiveness, and assessment into one department. 
These departments were generally led by an individual 
with a PhD who was a full member of the college’s 
leadership team, and employed sufficient staff to handle 
research above what is required for compliance. 



Building a culture of evidence and an institutional research 
function to support it can take a long time. At one college 
in our case studies that was advanced in its use of 
data for improving student outcomes and institutional 
performance, efforts in this direction began in the 1980s 
and took hold during the 1990s. Administrators at the 
college say that it has taken them ten to fifteen years since 
then to establish a culture in which data-based decision 
making is institutionalized. During that time, each new 
president of the college worked to preserve and extend 
gains made in terms of data collection, analysis, and 
college-wide research literacy. 

The importance of college leadership to making the 
necessary investment in IR capacity and promoting 
the needed changes in organizational practice and 
culture further suggests that present and future college 
presidents need to learn about and embrace strategies 
and methods for using data to continuously improve the 
impact of programs and services on student success. 
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